feat: per-folder RSS feeds, cont'd #866

bfahrenfort · 2024-02-14T03:29:41Z

Follow-up on #529 and accidental #865. @aarnphm

aarnphm

few notes. cc @jackyzha0 when you have bandwidth.

aarnphm · 2024-02-14T04:55:45Z

quartz.config.ts

@@ -70,6 +70,7 @@ const config: QuartzConfig = {
      Plugin.ContentIndex({
        enableSiteMap: true,
        enableRSS: true,
+        feedDirectories: ["index"], // For a feed for only pages in content/Folder/, add "Folder" to the array


I think enableRSS should also handle this, need to set custom feedDirectories.

Are there cases where you don't want to set RSS for folder?

I think that would be duplicative. Enable should toggle the whole state, with optional finer-grained control through a list.

I mean we should have enableRSS, and it will do recursive feed for folders, no need to expose this options to users.

For more fine-grained control users can modify the code themselves.

Or at least, we don't show this feedDirectories to quartz.config.ts.

The whole point of this PR is to expose RSS generation per-directory for easy configuration without having to modify the code of the plugin. The web is woefully underdeveloped when it comes to RSS and making it more accessible is a big goal of mine.

What's the problem with just generating RSS recursively with enable RSS?

My point is that if users want more fine grain per directory, they can do that themselves. By default, we just traverse the vault and generate them, right?

Not sure if there is something that I miss here.

also this is my two cent, probably @jackyzha0 would have better input than me here.

Ugh, if they set enable RSS to false, yet we still generate RSS per folder, wouldn't that be confusing?

ohh, the issue here is a disconnect in what a feed is. some have multiple purposes, but they're only emitted as .xml RSS feeds if enableRSS is on. Try the PR with enableRSS: false in your config. I named the map feedIndices because they're each a content index which might be a feed (but yeah, love that there are three meanings of index in this codebase haha).

L163: if (opts?.enableRSS)
feeds are only written as files if RSS is enabled. Regardless, we still need to generate them because the feed at feedIndices.get("index") is used as the sitemap and becomes the simplifiedIndex later. (note that "index" is referring to index.xml and not index.md; in this context it means "every page on the site" not "the homepage")

Then I suggest that for the index, we make a separate path, since contentIndex will be populated in downstream components.

Then I suggest that for the index, we make a separate path, since contentIndex will be populated in downstream components.

Not sure what you mean by this. what are you referring to with "index"? "index" as a key of feedIndices is going to have the feed for all pages on the site, index.xml will be an RSS feed for the content of "index" (only emitted if enableRSS is on), and the index contentIndex is used to generate the sitemap and populate simplifiedIndex.

"index" is part of feedIndices because that made the most sense with the structure of this file (and it means I only need one loop to generate all feeds, and it makes configuration by the user easier).

hmm you are right.

quartz/plugins/emitters/contentIndex.ts

bfahrenfort · 2024-02-14T16:37:03Z

Suggestions added. I'm upset that the idiomatic way to concatenate in-place is to expand the second array in a push call, but that can't be helped.

quartz/plugins/emitters/contentIndex.ts

bfahrenfort · 2024-02-14T19:49:44Z

I think the lint config is broken, it's wanting to add trailing commas to arguments of single-argument functions.

jackyzha0

i actually have a really hot take on this, curious to hear your opinion @bfahrenfort ...

we should generate an RSS feed per directory UNLESS that directory has an index.md to has noFeed: true in the frontmatter

imo, we should do content-level config within frontmatter wherever possible but I'm curious but what you think here

bfahrenfort · 2024-03-10T04:19:53Z

we should generate an RSS feed per directory UNLESS that directory has an index.md to has noFeed: true in the frontmatter

I worry about build times, doing this recursively could get clunky for large gardens. The main reason I wanted to retain the initial functionality and then opt-in to my new feature is because the current type is what's expected from effectively any other RSS integration on any site/SSG. We also don't have the dynamicity to do caching like Mastodon does for their per-tag and per-user feeds. Can maybe add a plugin option for this behavior? global-recursive: "disabled" | "directories" | "content" | "all" that ignores includeDirectories and instead watches excludeDirectories when set?

I definitely think just adding .xml to the link to the folder/file you're watching would be really cool UX but I only see it on projects that require webservers for a reason.

imo, we should do content-level config within frontmatter wherever possible but I'm curious but what you think here

Can absolutely add this! Would an rss: true in frontmatter work? Then I just filter the root contentindex with that

jackyzha0 · 2024-03-10T06:54:34Z

i think it shouldn't be too chunky, the processing for rss feeds is really light compared to what the rest of Quartz does so it would be dominated by the disk write speed (i.e. Hugo has this behaviour)

we can also reuse nestings, e.g. if we have

[content]

a
- b
- c
d
- e
- f

the rss feed for content would just be the feed of a + d

bfahrenfort · 2024-03-10T13:10:43Z

Reuse is fascinating because that basically just needs a DFS for generation then. Can I safely do recursion here?

jackyzha0 · 2024-03-10T16:52:22Z

you could use memoization here for the mapping of directory to rss feed and only compute if it hasnt already been computed (i think we shouldnt need recursion here)

bfahrenfort · 2024-03-10T21:08:05Z

Ooh, any resources on that? I'll add it when I get time.

gustavo-depaula · 2024-09-20T15:53:32Z

Is this blocked? Anyway we can help?

I really like these ideas:

we should generate an RSS feed per directory UNLESS that directory has an index.md to has noFeed: true in the frontmatter

the rss feed for content would just be the feed of a + d

I think this is the ideal, use top-level folders to do RSS channels

gustavo-depaula · 2024-09-20T19:55:55Z

cc/ @jackyzha0

aarnphm · 2024-09-20T20:36:12Z

Hi there, please don't ping maintainers as we all have real life and jobs to tend to. We will get to this at our own time.

If you want this then feel free to contact the author to continue their works.

Hopefully you would understand this

bfahrenfort · 2024-09-20T21:57:57Z

Hi @gustavo-depaula ,

I think I figured out how to implement it, I'm just waiting on finding the time to do so. My idea is to build a tree of indices that matches the directory structure, and then implement foldr to discard the shape of each subtree and emit the result.

bfahrenfort · 2024-11-08T03:55:05Z

Looks like I can implement something called a visitable tree for this.

Vet my pseudocode:

Turn the emitter plugin's content list into something viewable as a tree
generate an RSS entry for every leaf of this tree (effectively map)
Walk the tree of feeds bottom-up to compose a feed for each directory from the feeds and entries inside it, unless this directory has noFeed: true in its index.md frontmatter; and push this feed as a future for emission
await all emissions

I'm planning to emit feeds in the same directory as the directory it tracks, that way you can just add .rss to the end of the folder's link to get the feed.

jackyzha0 · 2024-11-08T17:21:41Z

Looks like I can implement something called a visitable tree for this.

Vet my pseudocode:

Turn the emitter plugin's content list into something viewable as a tree

generate an RSS entry for every leaf of this tree (effectively map)

Walk the tree of feeds bottom-up to compose a feed for each directory from the feeds and entries inside it, unless this directory has noFeed: true in its index.md frontmatter; and push this feed as a future for emission

await all emissions

I'm planning to emit feeds in the same directory as the directory it tracks, that way you can just add .rss to the end of the folder's link to get the feed.

yes! this was the implication of one of my earlier reviews :) makes a lot of sense, go for it

bfahrenfort · 2024-11-12T13:19:08Z

Revised implementation pseudocode:

translate ProcessedContent[] to a tree of directories and leaf files:
- initialize directories to empty Feeds (no Feed is created/will be generated unless the directory actually has content files)
- parse ContentDetails from files
- make sure the directory Feed has access to its enclosed index.md's metadata, if that file exists
(optimized out) ~~map() over tree to translate content to metadata~~
walk the tree bottom-up to:
- generate RSS text entry lists for each level, excluding files which have specified noRSS: true
- lift the entries from lower levels into the current level
- emit a feed subdirectory.rss for each level (which respects sort order and rssLimit), except for directories with noRSS: true in their index.md
emit index.xml
await all emissions

(done) ~~I'll work on collapsing the list-to-tree and map-tree steps into one pass next;~~ not thrilled about still needing two passes, but such is the nature of typescript.

bfahrenfort · 2024-11-21T00:29:06Z

Will squash for cleanliness when it's ready for merge, just let me know if there are any changes I need to make on review.

aarnphm

just some tiny styling change. cc jacky for final say

quartz/plugins/emitters/contentIndex/indexTree.ts

aarnphm · 2024-11-21T01:19:40Z

quartz/plugins/emitters/contentIndex.ts

+      // bfahrenfort: If I could finagle a Visitor pattern to cross
+      //  different datatypes (TransformVisitor<T, K>?), half of this pass could be
+      //  folded into the FeedGenerator postorder accept
+      const detailsOf = ([tree, file]: ProcessedContent): ContentDetails => {


Seems like this only concerns all folders that contains that is reachable by Quartz

so non-empty folders that doesn't have a index.md won't have a RSS file ig.

Seems like this only concerns all folders that contains that is reachable by Quartz

Yes.

so non-empty folders that doesn't have a index.md won't have a RSS file ig.

No, non-empty folders will all have RSS files, unless they have noRSS: true in the frontmatter of an index.md in the folder. detailsOf is called once per content file in the content array passed to emit().

Example:

content/ ├─ afolder/ ├─ bfolder/ │ ├─ a.md │ ├─ b.md ├─ cfolder/ │ ├─ index.md │ ├─ c.md ├─ index.md

produces:

public/ ├─ bfolder/ │ ├─ a.html │ ├─ b.html │ ├─ index.html (default) ├─ bfolder.rss (a, b) (except any that said noRSS) ├─ cfolder/ │ ├─ index.html (user-defined) │ ├─ c.html ├─ cfolder.rss (c, cfolder/index) (unless cfolder/index.md said noRSS) ├─ index.html ├─ index.xml (index, a, b, c, cfolder/index) (except any that said noRSS) ├─ sitemap.xml ├─ static/ │ ├─ contentIndex.json

aarnphm reviewed Feb 14, 2024

View reviewed changes

quartz/plugins/emitters/contentIndex.ts Outdated Show resolved Hide resolved

quartz/plugins/emitters/contentIndex.ts Outdated Show resolved Hide resolved

quartz/plugins/emitters/contentIndex.ts Outdated Show resolved Hide resolved

bfahrenfort force-pushed the SplitRSS branch from ccbd6cc to a674a6e Compare February 14, 2024 19:47

bfahrenfort requested a review from aarnphm March 7, 2024 20:58

jackyzha0 reviewed Mar 10, 2024

View reviewed changes

bfahrenfort force-pushed the SplitRSS branch from a9f87c0 to 86ef225 Compare November 12, 2024 13:06

bfahrenfort requested a review from jackyzha0 November 21, 2024 00:28

aarnphm approved these changes Nov 21, 2024

View reviewed changes

feat(contentIndex): Per-folder RSS feeds

41d2b6e

bfahrenfort force-pushed the SplitRSS branch from 8a0e78b to 41d2b6e Compare November 24, 2024 00:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: per-folder RSS feeds, cont'd #866

feat: per-folder RSS feeds, cont'd #866

bfahrenfort commented Feb 14, 2024

aarnphm left a comment

aarnphm Feb 14, 2024

bfahrenfort Feb 14, 2024

aarnphm Feb 14, 2024

bfahrenfort Feb 14, 2024

aarnphm Feb 14, 2024

aarnphm Feb 14, 2024

bfahrenfort Feb 14, 2024 •

edited

Loading

aarnphm Feb 14, 2024

bfahrenfort Feb 14, 2024

aarnphm Feb 14, 2024

bfahrenfort commented Feb 14, 2024

bfahrenfort commented Feb 14, 2024

jackyzha0 left a comment

bfahrenfort commented Mar 10, 2024 •

edited

Loading

jackyzha0 commented Mar 10, 2024

bfahrenfort commented Mar 10, 2024

jackyzha0 commented Mar 10, 2024

bfahrenfort commented Mar 10, 2024 •

edited

Loading

gustavo-depaula commented Sep 20, 2024

gustavo-depaula commented Sep 20, 2024

aarnphm commented Sep 20, 2024

bfahrenfort commented Sep 20, 2024

bfahrenfort commented Nov 8, 2024 •

edited

Loading

jackyzha0 commented Nov 8, 2024

bfahrenfort commented Nov 12, 2024 •

edited

Loading

bfahrenfort commented Nov 21, 2024

aarnphm left a comment

aarnphm Nov 21, 2024

bfahrenfort Nov 21, 2024 •

edited

Loading

feat: per-folder RSS feeds, cont'd #866

Are you sure you want to change the base?

feat: per-folder RSS feeds, cont'd #866

Conversation

bfahrenfort commented Feb 14, 2024

aarnphm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bfahrenfort Feb 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bfahrenfort commented Feb 14, 2024

bfahrenfort commented Feb 14, 2024

jackyzha0 left a comment

Choose a reason for hiding this comment

bfahrenfort commented Mar 10, 2024 • edited Loading

jackyzha0 commented Mar 10, 2024

bfahrenfort commented Mar 10, 2024

jackyzha0 commented Mar 10, 2024

bfahrenfort commented Mar 10, 2024 • edited Loading

gustavo-depaula commented Sep 20, 2024

gustavo-depaula commented Sep 20, 2024

aarnphm commented Sep 20, 2024

bfahrenfort commented Sep 20, 2024

bfahrenfort commented Nov 8, 2024 • edited Loading

jackyzha0 commented Nov 8, 2024

bfahrenfort commented Nov 12, 2024 • edited Loading

bfahrenfort commented Nov 21, 2024

aarnphm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bfahrenfort Nov 21, 2024 • edited Loading

Choose a reason for hiding this comment

bfahrenfort Feb 14, 2024 •

edited

Loading

bfahrenfort commented Mar 10, 2024 •

edited

Loading

bfahrenfort commented Mar 10, 2024 •

edited

Loading

bfahrenfort commented Nov 8, 2024 •

edited

Loading

bfahrenfort commented Nov 12, 2024 •

edited

Loading

bfahrenfort Nov 21, 2024 •

edited

Loading